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ABSTRACT 


This report examines what the National Assessment of Educational Progress (NAEP) can tell us 
about charter school enrollment and student performance compared to that of regular public 
schools. The study uses NAEP reading and mathematics data from 2011 and the earlier years 
when charter school data first became available (2003 for grade 4; 2005 for grade 8). The study 
finds that while charter schools still account for a relatively small percentage of overall public 
school enrollment (about 3%), this percentage is growing rapidly, particularly in large cities. A 
sizable jump was found in charter school enrollment for the black student subgroup. 

In terms of achievement, there is a consistent pattern of higher average NAEP scores for regular 
public schools than for charters when we look at the nation as a whole. However, the closer we 
focus in on large cities, where most charter schools are located, the more the picture changes in 
favor of charter schools. In all large cities combined, student achievement is roughly even 
overall, but the black and Hispanic subgroups show higher scores in charter schools. When we 
examined four urban areas specifically (DC, Atlanta, Chicago, Milwaukee), students in charter 
schools significantly outperformed their peers in regular public schools in many of the 
subjects/grades analyzed. Charter school class sizes are smaller, and there is some evidence that 
charter school students receive more instructional time in some subjects. The report recommends 
making some changes to the NAEP Data Explorer interface and including data for research 
purposes in the Trial Urban District Assessment (TUDA) database that is representative of all 
charter schools located within each participating urban district. 
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EXECUTIVE SUMMARY 


Charter schools are independent public schools that operate with a large measure of autonomy 
from local districts. Students attend charter schools by choice. The schools themselves are 
exempt from many state and local rules but are subject to accountability standards established in 
their charters. 

There has been considerable controversy over charter schools and whether they foster higher 
student achievement. In 2005, the National Center for Education Statistics (NCES) issued a 
study of America’s charter schools based on the 2003 NAEP reading and mathematics 
assessments for 4th grade. ^ The study found that students in charter schools were generally 
lagging behind their counterparts in regular public schools. But the differences were statistically 
significant only for math, not for reading. Given the growth of charter schools since 2003, it is 
important to take a fresh look at what NAEP can tell us about charter school enrollment and 
student performance, based on 201 1 data. 

This report examines five questions using NAEP data: 

• Who attends charter schools? How has charter school enrollment changed over the past 
decade? 

• Does student achievement differ between charter schools and regular public school 
students? Has that changed since 2003? 

• How do certain subgroups, such as low-income, black, and Hispanic students, perform in 
charter schools compared to regular schools? 

• How does the performance of charter schools compare to that of regular public schools 
located within the same city? 

• Are there important differences between charter and regular schools that are captured by 
the NAEP background data? 

NAEP collects a rich set of student, teacher, and school responses to background questionnaires 
(referred to as “background variables”) that can help in understanding the context for NAEP 
achievement results and give insights into how to improve them. We can use NAEP to address 
the charter school questions above because one of the background variables that has been 
collected since 2003 from each school in the NAEP sample is whether it is a charter or regular 
public school. 


^ For the 2003 charter school study, NAEP oversampled charter schools at grade 4 to make sure that the sample of 
charter school students was large enough to conduct the necessary analyses. In 2005, NAEP started assessing 
larger numbers of students and from that time forward, the NAEP sample Included a representative sample of 
charter schools without having to oversample. 
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NAEP background questions are a potentially important but largely underused national resource 
(Smith et al., 2012). This is one of several exploratory analyses, commissioned by the National 
Assessment Governing Board (NAGB) to illustrate the usefulness of NAEP background data for 
helping to understand important education issues. However, there are limitations to this study. 
One cannot conclude from the results the causal effects of attending a charter school on student 
achievement. The study provides descriptive and correlational evidence and by itself does not 
prove causation. 


Who Attends Charter Schools 


The National Center for Education Statistics (NCES, 2010) reports that from 2000 to 2010,^ the 
number of students enrolled in public charter schools more than quadrupled— from 0.3 million to 
1.6 million students. During this period, the percentage of all public schools that were charter 
schools increased from 2 to 5 percent, numbering 5,000 schools in 2010. At that time, about 55% 
of charter schools were located in cities, 21% were in suburban areas, 8% were in towns, and 
16% were in rural areas. In some cities, charter schools serve a large proportion of students. Eor 
instance, in New Orleans more than half of all public school students attend charter schools, and 
in the District of Columbia, more than a third. 

Questions have been raised about whether charter schools are appropriately serving students with 
disabilities. In response, the U.S. Government Accountability Office (GAO, 2012) conducted a 
study using federal data and found that charter schools enrolled a lower percentage of students 
with disabilities than traditional public schools. In school year 2009-2010, they found that 
approximately 11% of students enrolled in traditional public schools were students with 
disabilities compared to about 8% of students enrolled in charter schools. 

This study, using NAEP 2011 data, finds that charter schools are playing an increasingly 
important role in American education, particularly in large cities and for certain subgroups. 
Charter school data go back to 2003 in grade 4; 2005 in grade 8; and are only available for 2009 
in grade 12. 

The main findings related to charter school enrollment are: 

• Nationally, students attending charter schools account for a small slice of overall public 
school enrollment: about 3% in 2011 at grades 4 and 8. Still, this represents a significant 
increase, compared to 1% in 2003/05. 

• Higher percentages of students living in large cities attend charter schools. Eor example, 
in grade 4, charter schools enrolled 3% of all large city students in 2003 and this grew to 
6% in 2011. 

• While all of the subgroups analyzed showed significant increases in charter school 
enrollment since 2003, the most notable jump was for the black subgroup. Eor example. 


^ 2010 is the most current year for which NCES charter school enrollment data are available. 
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in the large cities, the percentage of grade 4 black students attending charter schools grew 
from 4% in 2003 to 12% in 2011. The percentage of black low-income students attending 
charter schools in the large cities is roughly similar to that of black students in general. 

• There are also differences in the student compositions of charter and regular public 
schools. Both nationally and in the large cities, charter student bodies include a 
significantly larger proportion of black students than regular public schools. Regular 
schools have significantly larger proportions of white students. 

• In grade 4, a larger proportion of the charter school student body was low-income in 
2011, compared to regular schools. A similar pattern can be seen at grade 8, but the 
difference between charter and regular schools is not significant. 

• In the large cities, in grade 4, regular schools had a significantly larger percentage of 
Hispanic students in their student bodies in 201 1. In grade 8 the same was true in 2005, 
but by 201 1, the Hispanic composition of the two types of schools became more even. 

• Also in the large cities in 201 1, regular schools had significantly higher rates of 
enrollment for students with disabilities at grade 8, and English language learners at 
grade 4. 


Achievement of Charter School Students 


An earlier NCES study (2005) comparing student achievement in charter and non-charter public 
schools using 2003 NAEP data, found that charter school performance lagged behind that of 
regular schools in 4* grade math, and was about the same in 4* grade reading. That study noted 
that when comparing the performance of charter and regular public school students, it is 
important to compare students who share a common characteristic. Eor example, in math, 4* 
grade charter school students as a whole did not perform as well as their public school 
counterparts. However, the math performance of white, black, and Hispanic 4* graders in charter 
schools was not measurably different from the performance of 4* graders with similar 
racial/ethnic backgrounds in other public schools. 

More recently, Betts and Tang (201 1) conducted a meta- analysis of several rigorous studies that 
used either experimental (lottery) or student-level growth based methods to infer the causal 
impact of attending a charter school on student performance. Erom this set of studies, Betts and 
Tang concluded that charter elementary schools, on average, outperformed traditional public 
schools in reading and in math, and charter middle schools were doing better in math. Their 
analysis found no substantial difference in math or reading achievement between traditional and 
charter high schools or in reading achievement at the middle school level. 

The main findings from our analysis of NAEP achievement data are: 
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• At the national level, there is a consistent pattern of higher average NAEP scores for 
regular public schools than for charter schools. This pattern is apparent in all 
grade/subjects analyzed: grades 4, 8 and 12 in reading, math, and science. 

• NAEP scores in grades 4 and 8 reading and math have increased between 2003/05 and 
201 1, in both regular public and charter schools, with larger gains for charter schools. 

The gains for regular public schools tend to be statistically significant, while a similar 
amount of growth for charter schools does not, probably because of the small charter 
school sample size. 

• Eocusing on large cities, average NAEP scores for charter and regular schools were 
similar, both in 2003/05 and 2011. The only significant difference was in grade 4 math in 
2003, in favor of regular public schools. By 201 1 this difference had disappeared. 

• However, the findings tend to favor charter schools when one focuses on black, Hispanic, 
and low-income students within the large cities. In many subject/grade combinations 
students in these subgroups in charter schools performed significantly better in 201 1 than 
those in regular public schools. By contrast, in 2003/2005 these subgroups performed 
similarly in charter and regular schools, and in one case (low-income students in grade 4 
math), the regular schools were ahead. 

• The performance of black low-income students attending charter schools in large cities is 
particularly striking. This group has shown a large increase in scores. In 201 1 their 
achievement was significantly higher than that of similar students in regular large-city 
schools in grade 8 reading and grades 4 and 8 math. 

• In the large cities, the only significant subgroup findings in favor of regular schools in 
2011 were for Asians (in grade 4 math) and whites (in grade 4 reading). 

• When we look more closely at a few large urban districts, the 2011 results clearly favor 
charter schools. In the four cities where NAEP data permitted comparisons (DC, Atlanta, 
Chicago and Milwaukee), students in charter schools significantly outperformed their 
peers in regular public schools in many of the subjects/grades analyzed. In those four 
districts, there are no subjects/grades where regular schools significantly outperformed 
charter schools. 

Why do we find that charter schools underperform regular public schools nationally, but 
outperform them in large cities? One possible explanation is that at the national level, NAEP 
results for regular public schools include a wide range of schools, including many high- 
performing suburban schools in high-income communities. If the purpose is to compare charter 
and regular schools, the fairest way is to compare them within similar locations. The location 
where charter school enrollment is most concentrated is the large cities. As noted above, 6% of 
large city grade 4 students and 8% of large city grade 8 students attend charter schools, 
compared to 3% at each grade nationally. Making comparisons within large cities increases the 
likelihood that the regular and charter schools are serving similar populations of students. 
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Other Factors Related to Instruction 


There are hundreds of variables collected from students, teachers, and school administrators each 
time NAEP is administered and very little of those data get used by researchers. For this 
exploratory study, we compared charter and regular public schools on a few policy-relevant 
factors (time spent on core subjects, class size, and teacher qualifications), to give a flavor of the 
kinds of information that are potentially available. These analyses were run for grades 4 and 8 in 
large cities. 

The main findings are that in the large cities: 


• Grade 8 charter school students get significantly more time per week of language arts 
instruction. Grade 8 charter school students may also be getting more instruction in math, 
although the finding is not significant. 

• Class sizes tend to be larger in regular schools than in charter schools. 

• Significantly more teachers in regular public schools have a major in the subject they 
teach. 

• More grade 8 math teachers in charter schools entered the teaching profession through an 
alternate certification program. 

• Teachers in regular schools have more years of teaching experience than teachers in 
charter schools. 


Strengthening NAEP Data and the Data Explorer 

The main way that members of the public and education researchers access NAEP data is 
through the online NAEP Data Explorer. It is a powerful tool and, with practice, researchers can 
conduct a wide variety of analyses. However, the interface is not as user-friendly as it could be. 
One of the purposes of this exploratory study was to identify ways that the NAEP background 
data could be improved. We offer these suggestions based on our experience: 

• Refine the online Data Explorer to make it as intuitive and efficient possible. 

• Find a better way to search the variables and filter those available in a given year. 

• Take stock of background variables that may no longer be relevant and consider 
reinstating some that are still of interest. 


Charter schools are likely to continue to grow in importance. They surely will be the focus of 
future research. These schools are serving ever-larger numbers and percentages of students — 
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particularly minority and low-income students. The large urban districts are where studies of 
charter schools should be focused. So finally we suggest: 

• Find a way to include results for all charter schools within the geographic boundaries of 
an urban district in the TUDA dataset, for research purposes. 
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Who Attends Charter Schools and 
How Are Those Students Doing? 

DATA AND FINDINGS 


I. Who Attends Charter Schools? 

Proportion of Public School Students Attending Charter Schools 

How large of a role do charter schools play in American public education? The percentages of 
all public school students attending charter schools are shown in table la. These figures are 
based on the NAEP mathematics sample in each grade. Charter school data go back to 2003 in 
grade 4; 2005 in grade 8; and are only available for 2009 in grade 12. 

The main findings are: 

• Students attending charter schools account for a small slice of overall public school 
enrollment: just 3% in 2011. However, the percentage of public school students attending 
charter schools has increased significantly over the past decade. In grade 4, the 
percentage rose from 1% in 2003 to 3% in 2011, and the same was true in grade 8 from 
2005 to 2011. 

• While all of the subgroups analyzed showed significant increases in charter school 
enrollment, the most notable jump was for the black subgroup. In grade 4, the percentage 
of black public school students attending charter schools increased from 2% in 2003 to 
7% in 2011. In grade 8, the percentage of black students enrolled in charter schools rose 
from 3% in 2005 to 6% in 2011. 
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Table 1a. National percentage of public school students attending charter schools, by grade and 
subgroup, for earliest year in which charter school data are available and 201 1 (based on NAEP 
mathematics sample) 



National Grade 4 

National Grade 8 

National 
Grade 12 


2003 

2011 

2005 

2011 

2009 

All Students 

1 

3* 

1 

3* 

2 

White 

1 

2* 

1 

2* 

1 

Black 

2 

7* 

3 

6* 

4 

Hispanic 

1 

3* 

1 

4 

3 

Asian/Pacific 

Islander 

1 

2* 

1 

2 

1 

Students with 
disabilities 

1 

2* 

2 

3* 

3 

English 

language 

learner 

1 

2* 

2 

3 

5 

Low-income 
(free/ reduced 
price lunch) 

1 

3* 

2 

3* 

3 


*The difference between base year and 201 1 is statistically signi 


'leant at the .05 level. 


Most charter schools are located in urban areas, so it is not surprising that charter schools serve 
larger proportions of students in large cities than is the case nationally, as shown in table lb. 
Grade 12 results are not shown because they cannot be disaggregated for large cities. For this 
analysis, we further disaggregated the data for the black low-income group. 

The main findings regarding charter school enrollment in the large cities are: 

• The percentage of public school students attending charter schools in the large cities has 
grown significantly. In grade 4, charter schools enrolled 3% of all large city students in 
2003 and this grew to 6% in 201 1. In grade 8, charter schools enrolled 5% of students in 
2005 which grew to 8% in 201 1. 

• There has been an especially large increase in the percentage of black students attending 
charter schools in large cities. In 2003, 4% of black public schools students in large cities 
attended charter schools; in 201 1 that figure rose to 12%. In grade 8 the percentage went 
from 8% in 2005 to 13% in 2011. 

• The percentage s of black low-income students attending charter schools in large cities 
are roughly similar to that of black students in general. Again, there was significant 
growth in charter school enrollment for this subgroup between 2003 and 2011. 
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Table 1b. Percentage of large city students attending charter schools, by grade and subgroup, 
for earliest year in which charter school data are available and 201 1 (based on NAEP 


mathematics sampi 

e) 


Large Cities Grade 4 

Large Cities Grade 8 


2003 

2011 

2005 

2011 

All Students 

3 

6* 

5 

8* 

White 

2 

5* 

4 

5 

Black 

4 

12* 

8 

13* 

Hispanic 

2 

4* 

3 

8 

Asian/Pacific 

Islander 

# 

2* 

2 

6 

Students with 
disabilities 

2 

5* 

5 

6 

English language 
learner 

2 

3* 

3 

9 

Low-income (free/ 
reduced price 
lunch) 

2 

6* 

4 

9* 

Black low-income 

3 

11* 

5 

12* 

* The difference be 

tween base year and 201 1 is statistically significant at the .05 1 


# Rounds to 0 


Student Composition of Charter Schools Compared to Regular Public Schools 

Tables Ic and Id provide another perspective on who attends charter schools. These tables show 
differences in the student composition of charter and regular public schools, addressing questions 
such as: Is the student body at charter schools made up of larger or smaller percentages of low- 
income students than at regular public schools? 

The main findings from table Ic are: 

• Nationally, charter student bodies include significantly larger proportions of black 
students than regular public schools. Regular public schools have larger proportions of 
white students. This was true in grades 4 and 8 in 2003/05, and was still the case in 2011. 

• In grade 4, charter student bodies included a significantly larger percentage of low- 
income students in 201 1. A similar pattern can be seen at grade 8, but the difference 
between charter and regular schools is not significant. 
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Table 1c. At the national level, percentages of students attending charter and regular schools 
that fall into various subgroups (based on NAEP mathematics sample) 



National Grade 4 

National Grade 8 


2G 

103 

2G 

11 

2G 

05 

20 

11 


Charter 

Regular 

public 

Charter 

Regular 

public 

Charter 

Regular 

public 

Charter 

Regular 

public 

White 

45 

58* 

35 

53* 

40 

60* 

34 

54* 

Black 

31* 

17 

37* 

15 

37* 

16 

32* 

15 

Hispanic 

20 

19 

22 

24 

17 

17 

28 

22 

Asian/Pacific 

Islander 

2 

4* 

3 

5 

3 

5 

4 

6 

Students with 
disabilities 

8 

11* 

9 

12 

13 

10 

10 

11 

English 

language 

learners 

9 

9 

9 

11 

6 

6 

7 

6 

Low-income 
(free/ reduced 
price lunch) 

42 

44 

59* 

52 

45 

39 

57 

47 


*The difference between the percentages for charter and regular public schools is statistically 
significant at the .05 level. 


The findings for large cities, displayed in table Id, are: 

• As with the national picture, large city charter schools have larger proportions of black 
students, although at grade 8 in 2011, the difference is not statistically significant. 

• In grade 4, regular schools in the large cities had a significantly larger percentage of 
Hispanic students in their student bodies in 2011. In grade 8 the same was true in 2005, 
but by 201 1, the Hispanic composition of the two types of schools became more even. 

• In the large cities, regular public schools had a higher rate of enrollment for students with 
disabilities in 2011, and at grade 8, the difference was significant. 

• In large cities, regular public schools had significantly larger proportions of English 
language learners at grade 4 in 2003 and 2011. 
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Table 1d. In large cities, percentages of students attending charter and regular schools that fall 
into various subgroups (based on NAEP mathematics sample) 



Large Cities Grade 4 

Large Cities Grade 8 


2003 

2G 

11 

2G 

05 

20 

11 


Charter 

Regular 

public 

Charter 

Regular 

public 

Charter 

Regular 

public 

Charter 

Regular 

public 

White 

17 

22 

16 

20 

22 

24 

12 

21* 

Black 

51* 

33 

53* 

25 

53* 

31 

41 

25 

Hispanic 

29 

36 

27 

45* 

20 

36* 

41 

44 

Asian/Pacific 

Islander 

1 

7* 

2 

8* 

4 

8* 

5 

9 

Students with 
disabilities 

8 

10 

9 

11 

11 

10 

8 

11* 

English 

language 

learner 

13 

20* 

12 

22* 

9 

13 

12 

11 

Low-income 
(free/ reduced 
price lunch) 

67 

69 

73 

74 

49 

63* 

72 

69 

Black low- 
income 

35 

28 

41* 

22 

26 

23 

31* 

20 


*The difference between the percentages for charter and regular public schools is statistically 
significant at the .05 level. 
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II. Student Achievement in Charter and Regular Public Schools 

This section begins with a comparison of NAEP achievement in charter and regular public 
schools at the national level. We then focus in on performance in the large cities, the location 
where charter school enrollment is most concentrated. Finally, we compare achievement in 
charter and regular schools in a few particular urban districts where NAEP data permitted 
comparisons. 

Achievement for All Students 


Are students in charter schools reaching higher levels of achievement than students in regular 
public schools? Table 2a presents the results for students overall in grades 4 and 8, and table 2b 
shows the results for grade 12. The main findings: 

• At the national level, there is a consistent pattern of higher average NAEP scores for 
regular public schools than for charter schools. This pattern is apparent in all 
subject/grade combinations and years analyzed. In some cases, such as grade 4 math, the 
regular schools’ scores are significantly higher. 

• Grade 12 math and reading scores were significantly higher in regular public schools than 
in charter schools in 2009, the only year for which data are available. 

• In science, regular schools also posted significantly higher scores at all grade levels in 
2009. 

• NAEP scores in grades 4 and 8 reading and math have increased between 2003/05 and 
201 1, in both regular public and charter schools, with larger gains for charter schools. 

Not shown in table 2a is that the gains for regular public schools tend to be statistically 
significant, while a similar amount of growth for charter schools does not, probably 
because of the small charter school sample size. 


Table 2a. National NAEP achievement in charter and regular public schools, for earliest year in 
which charter school data are available and 201 1 



Nationai 200. 
2005 (g 

3 (grade 4) or 
rade 8) 

National 2011 

Cha 

2011 -2( 

nge 

303/2005 


Charter 

Reguiar 

Public 

Charter 

Regular 

Public 

Charter 

Regular 

Public 

Grade 4 
Reading 

212 

217 

218 

220 

6 

3 

Grade 8 
Reading 

255 

260* 

261 

264 

6 

4 

Grade 4 
Math 

228 

234* 

237 

240* 

9 

6 

Grade 8 
Math 

268 

278* 

281 

283 

13 

5 


*The difference between charter schools and regular public schools is statistically significant at 
the .05 level. 
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Table 2b. National grade 12 and science results for charte r and regular public schools, 2009 



National 2009 


Charter 

Regular Public 

Grade 12 Reading 

276 

287* 

Grade 12 Math 

138 

153* 

Grade 4 Science 

139 

149* 

Grade 8 Science 

141 

149* 

Grade 12 Science 

137 

149 


*The difference between charter schools and regular public schools is statistically significant at 
the .05 level. 

Focusing on the results for large eities (grade 12 data are not available), tables 2c and 2d show 
that: 

• Average NAEP seores for eharter and regular publie sehools in large eities generally 
similar, both in the base year and 2011. The only signifieant differenee was in grade 4 
math in 2003, in favor of regular publie sehools. By 2011, that difference disappeared. 

• Both eharter and regular sehools showed growth over the years in all subjeets/grades 
analyzed. 


Table 2c. Large city NAEP achievement in charter and regular public schools, for earliest year in 
which charter school data are available and 201 1 



Large Cities 2003 (grade 4) 
or 2005 (grade 8) 

Large Cities 201 1 

Cha 

2011 -2( 

nge 

D03/2005 


Charter 

Regular 

Public 

Charter 

Regular 

Public 

Charter 

Regular 

Public 

Grade 4 
Reading 

201 

204 

210 

211 

9 

7 

Grade 8 
Reading 

251 

250 

254 

255 

3 

5 

Grade 4 
Math 

216 

224* 

232 

233 

16 

9 

Grade 8 
Math 

264 

265 

275 

274 

11 

9 


*The difference between charter schools and regular public schools is statistically significant at 
the .05 level. 


Table 2d. Large city NAEP science results, 2009 



Large Cities 2009 


Charter 

Regular Public 

Grade 4 Science 

130 

135 

Grade 8 Science 

132 

134 
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Achievement by Subgroup 


Up to this point, results have been in favor of regular public schools. But as discussed in part I of 
this report, the populations of students who attend charter and regular public schools differ. 
Therefore, it is important, when comparing achievement across these two types of schools, to 
compare like groups to one another. Then one can address question such as: In large cities, are 
low-income students who attend charter schools performing better than their low-income peers in 
regular public schools? 

Tables 2e and 2f display achievement broken down by subgroup for reading and math in grades 
4 and 8, the subjects/grades with the most complete data. The main findings for subgroups at the 
national level are: 

• In several subject/grade combinations, a subgroup performed significantly better 
in regular public schools in 2003/05; but by 201 1 that subgroup performed 
similarly in both charter and regular schools. This was true of low-income 
students in grade 4 reading and math; and black students and students with 
disabilities in grade 8 math. 

• Hispanic students used to perform similarly in charter and regular schools in 
2003/05; but in 201 1 they performed significantly better in grade 4 reading and 
grade 8 math in charter schools. The same pattern emerged for ELLs in grade 4 
reading. 

• Asian students were the one subgroup that performed significantly better in 
regular public schools in 201 1, in grades 4 and 8 math. 

• In both charter and regular schools, almost all subgroups showed growth in scores 
from 2003/05 to 2011, in all subject/grade combinations. There were just a few 
instances where the English language learner or Asian subgroups showed 
declines. 
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Table 2e. National subgroup achievement in charter and regular public schools, for earliest year 
in which charter school data are available and 201 1 



National 2003 (grade 4) 
2005 (grade 8) 

National 201 1 

Char 

201 1 - 20 

ige 

03/2005 


Charter 

Regular 

Public 

Charter 

Regular 

Public 

Charter 

Regular 

Public 

Grade 4 Reading 

White 

227 

227 

232 

230 

5 

3 

Black 

195 

197 

206 

205 

9 

8 

Hispanic 

201 

199 

212* 

205 

11 

6 

Asian/Pacific Islander 

224 

225 

226 

234 

2 

9 

Students with disabilities 

180 

184 

184 

186 

4 

2 

English language 
learners 

183 

186 

195* 

188 

12 

2 

Low-income 
(free/reduced lunch) 

195 

201* 

208 

207 

13 

6 

Grade 8 Reading 

White 

271 

269 

274 

272 

3 

3 

Black 

239 

242 

251 

247 

12 

5 

Hispanic 

252 

245 

255 

251 

3 

6 

Asian/Pacific Islander 

268 

270 

272 

275 

4 

5 

Students with disabilities 

228 

226 

235 

230 

7 

4 

English language 
learners 

235 

224 

231 

223 

-4 

-1 

Low-income 
(free/reduced lunch) 

243 

247 

253 

251 

10 

4 

Grade 4 Math 

White 

242 

243 

251 

249 

9 

6 

Black 

214 

216 

224 

224 

10 

8 

Hispanic 

219 

221 

234 

229 

15 

8 

Asian/Pacific Islander 

248 

246 

243 

256* 

-5 

10 

Students with disabilities 

209 

214 

218 

218 

5 

4 

English language 
learners 

211 

214 

222 

219 

11 

5 

Low-income 
(free/reduced lunch) 

216 

222* 

229 

229 

13 

7 

Grade 8 Math 

White 

284 

288 

296 

293 

12 

5 

Black 

248 

254* 

266 

262 

18 

8 

Hispanic 

270 

261 

277* 

269 

7 

8 

Asian/Pacific Islander 

287 

295 

287 

303* 

0 

8 

Students with disabilities 

235 

244* 

254 

249 

19 

5 

English language 
learners 

251 

244 

254 

243 

3 

-1 

Low-income 
(free/reduced lunch) 

255 

262 

271 

269 

16 

7 

*The difference between c 

barter schools and regular public schools is statistically signi 

ficant at 


the .05 level. 
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The findings tend to favor charter schools the more we focus in on large cities and subgroups 
within the large cities. The main findings from Table 2f are: 

• In 201 1, the Hispanic subgroup performed significantly higher in charter schools than in 
regular schools. This was true in grade 4 reading, and in grades 4 and 8 math. 

• Black students in large cities performed similarly in charter and regular schools in the 
base year of our analysis (2003/05). But in 201 1, achievement for black students was 
significantly higher in charter schools in grade 8 reading and grade 4 math. 

• A similar pattern was found for the black- low-income subgroup. In 201 1, 
achievement for this group was significantly higher in charter schools in grade 8 
reading, and grades 4 and 8 math. The black low-income group in charter schools 
showed a striking increase in scores between 2003/05 and 2011. 

• The only significant findings in favor of regular schools in 201 1 were for the 
Asian subgroup (grade 4 math) and the white subgroups (grade 4 reading). 
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Table 2f. Large city subgroup achievement in charter and regular public schools, for earliest 


year in which charter school data are available and 20 1 1 



Large Cities 2003 
(gr 4) and 2005 (gr 8) 

Large Cities 201 1 

Change 

2011 -2003/2005 


Charter 

Reguiar 

Charter 

Reguiar 

Charter 

Reguiar 

Grade 4 Reading 

White 

220 

226 

225 

233* 

5 

7 

Black 

193 

193 

206 

201 

13 

8 

Hispanic 

202 

197 

210* 

203 

8 

6 

Asian/Pacific Islander 

t 

223 

211 

225 

t 

2 

Students with disabilities 

175 

175 

177 

177 

2 

2 

English language learners 

181 

184 

189 

187 

8 

3 

Low-income 
(free/reduced lunch) 

192 

197 

205 

203 

13 

6 

Black low-income 

189 

191 

203 

199 

14 

8 

Grade 8 Reading 



White 

269 

270 

273 

273 

4 

3 

Black 

241 

239 

250* 

244 

9 

5 

Hispanic 

253 

243 

251 

248 

-2 

5 

Asian/Pacific Islander 

t 

266 

269 

270 

t 

4 

Students with disabilities 

223 

213 

226 

221 

3 

8 

English language learners 

239 

220 

227* 

219 

-12 

-1 

Low-income (free/reduced 
lunch) 

243 

243 

250 

248 

7 

5 

Black low-income 

233 

237 

250* 

242 

17 

5 

Grade 4 Math 



White 

230 

243* 

249 

251 

19 

8 

Black 

210 

212 

227* 

221 

17 

9 

Hispanic 

217 

219 

232* 

228 

15 

9 

Asian/Pacific Islander 

t 

246 

231 

249* 


3 

Students with disabilities 

199 

204 

205 

210 

6 

6 

English language learners 

209 

211 

219 

219 

10 

8 

Low-income (free/reduced 
lunch) 

212 

217* 

228 

227 

6 

10 

Black low-income 

207 

210 

225* 

219 

18 

9 

Grade 8 

Math 



White 

286 

288 

292 

295 

6 

7 

Black 

250 

250 

267 

260 

17 

10 

Hispanic 

272 

258 

278* 

267 

6 

9 

Asian/Pacific Islander 

295 

289 

281 

297 

-6 

8 

Students with disabilities 

233 

230 

247 

238 

14 

8 

English language learners 

250* 

238 

256 

239 

6 

1 

Low-income 

256 

256 

272* 

265 

16 

9 

Black low-income 

244 

246 

265* 

257 

21 

11 


*The difference between charter schools and regular public schools is statistically sign! 


leant at 


the .05 level. 
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Achievement for Selected Urban Districts 


NAEP reports results for a number of large school districts, referred to as the Trial Urban District 
Assessment (TUDA). Due to a policy change in 2009, charter schools within the geographic 
boundaries of an urban district, but operated independently of the district and not included in the 
district’s Adequate Yearly Progress (AYP) report under NCLB, are no longer included in the 
NAEP results for TUDA districts. School districts vary in terms of whether the charter schools 
within their boundaries are included in the district’s AYP results. 


Table 2g presents results for several districts with charter school data available. Results are 
shown for all students and the low-income subgroup. Other subgroups (e.g., black, Hispanic) are 
not included because samples were not large enough to meet NAEP reporting standards. 


The findings are: 


• Within the four cities analyzed, the pattern is clear that students in charter schools 
significantly outperform their peers in regular public schools. 

• There was no subject/grade in any of the cities analyzed where regular schools scored 
significantly higher than charter schools. 


Why do charter schools underperform regular public schools nationally, but outperform them in 
large cities? At the national level, NAEP results for regular public schools include a wide range 
of schools, including many high-performing suburban schools in high-income communities. If 
the purpose is to compare charter and regular schools, the fairest way is to compare them within 
similar locations. As noted above, charter school enrollment is most concentrated in large cities: 
6% of large city grade 4 students and 8% of large city grade 8 students attend charter schools, 
compared to 3% at each grade nationally. Making comparisons within large cities increases the 
likelihood that regular and charter schools are serving similar populations of students. 


Table 2g. For selected urban districts, subgroup achievement in charter schools (CH) and 
regular public schools (RP) 



DC 

Atianta 

Chicago 

Miiwaukee 


CH 

RP 

CH 

RP 

CH RP 

CH 

RP 

Grade 4 Reading 

All students 

200 

201 

216 

211 

214* 

202 

214* 

193 

Low-income (free/reduced lunch) 

195* 

188 

207 

200 

213* 

199 

208* 

188 

Grade 8 Reading 

All students 

249* 

237 

264* 

252 

254 

253 

237 

238 

Low-income (free/reduced lunch) 

245* 

228 

262* 

246 

257* 

249 

234 

234 

Grade 4 Math 

All students 

222 

222 

233* 

228 

235* 

223 

233* 

218 

Low-income (free/reduced lunch) 

218* 

211 

223 

218 

234* 

220 

228* 

215 

Grade 8 Math 

All students 

267* 

255 

274* 

265 

269 

270 

257 

254 

Low-income (free/reduced lunch) 

262* 

246 

270* 

259 

270 

267 

255 

250 


*The difference between charter and regular public schools is statistically significant at .the 05 level. 
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III. Other School Factors 


In this section we compare charter and regular public schools on several school factors other than 
student achievement: time spent on core subjects, class size, and teacher qualifications. The 
NAEP background data also include many questions about instructional content and methods 
that are more subject- specific and are not addressed in this report. 

These analyses were conducted for large cities rather than at the national level because, as 
demonstrated in earlier parts of this report, that is where the majority of charter schools are 
located and it is more meaningful to compare charter and regular schools serving similar 
populations of students in similar locations. We do not present results for particular school 
districts, as we did in part II, because of insufficient data at the district level. 

Time Spent on Core Subjects 

Do charter and regular schools located in large cities differ in the amount of time that they spend 
on core subjects? Table 3a displays the 201 1 results for large cities: 

• A significantly higher percentage of regular school teachers reported spending 
less than five hours a week on grade 8 language arts. Overall, it appears that 
charter school students are getting more instruction in grade 8 language arts. 

• A significantly higher percentage of regular school teachers reported spending 5-7 
hours a week on grade 4 math. Higher percentages of charter school than regular 
teachers reported spending more than 7 hours a week in that subject but the 
difference is not significant. 

• There is suggestive evidence (large differences but not statistically significant) 
that charter school students are getting more instruction in grade 8 math. 


Table 3a. In large cities, percentages of teachers reporting various amounts of time spent per 
week on language arts and math instruction, 201 1 



Less than 5 hrs/week 

5-6.9 hrs/week 

7 or more hrs/week 


Charter 

Regular 

Public 

Charter 

Regular 

Public 

Charter 

Regular 

Public 

Grade 4 Language Arts 

11 

11 

13 

9 

78 

76 

Grade 8 Language Arts 

19 

35* 

47 

34 

34 

30 

Grade 4 Math 

13 

9 

39 

52* 

49 

40 

Grade 8 Math 

35 

46 

35 

37 

30 

17 


*The difference between charter schools and 
the .05 level. 


regu 


ar public schools is statistically significant at 
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Class Size 


The NAEP background questionnaire asks teachers how many students are in the class being 
assessed. Table 3b shows that: 

• Class sizes tend to be larger in regular schools than in charter schools. Significantly more 
regular school teachers reported having more than 26 students in their classes in grade 4 
language arts and grade 8 math. 


Table 3b. Percentages of teachers in large cities reporting various numbers of students in their 
classes, 201 1 



18 or fewer students 

19-25 students 

26 or more students 


Charter 

Regular 

Public 

Charter 

Regular 

Public 

Charter 

Regular 

Public 

Grade 4 Language Arts 

24 

11 

40 

42 

35 

47* 

Grade 4 Math 

20 

12 

42 

42 

38 

46 

Grade 8 Language Arts 

10 

9 

40 

29 

49 

62 

Grade 8 Math 

12 

7 

51 

32 

37 

61* 


*The difference between charter schools and 
the .05 level. 


regu 


ar public schools is statistically significant at 


Teacher Qualifications 

Table 3c presents findings regarding the qualifications of math teachers, and table 3d presents 
findings for English/language arts teachers: 


• In large cities, significantly more students in regular public schools have teachers with a 
major in the subject being taught, whether it is math (grade 4 and 8) or English/language 
arts (grade 8). 

• Significantly more grade 8 students in charter schools are taught by math teachers who 
entered the teaching profession through an alternate certification program. 

• Significantly more students in regular public schools have teachers with more than 10 
years of teaching experience than those in charter schools. Significantly more charter 
school students are taught by teachers with less than 4 years of teaching experience. This 
is true at grades 4 and 8, for both math and reading. 
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Table 3c. Percentages of students in large cities whose teachers have various qualifications, 
based on NAEP math teacher questionnaire, 201 1 



Grade 4 Math 

Grade 8 Math 


Charter 

Regular 

Public 

Charter 

Regular 

Public 

Alternate certification program 

18 

18 

47* 

30 

Certified by NBPTS 

12 

13 

10 

11 

Major in math 

4 

9* 

11 

21* 

Major in math education 

16 

21 

13 

17 

Years taught 

0-4 years 

47* 

23 

43* 

19 

5-9 years 

33 

33 

31 

30 

10-19 years 

16 

30* 

21 

33* 

20 - 1 - years 

3 

14* 

5 

18* 


*The difference between charter schools and regular public schools is statistically significant at 
the .05 level. 


Table 3d. Percentages of students in large cities whose teachers have various qualifications. 


based on NAEP reading teacher questionnaire, 201 1 



Grade 4 Reading 

Grade 8 Reading 


Charter 

Regular 

Public 

Charter 

Regular 

Public 

Alternate certification program 

23 

16 

26 

25 

Certified by NBPTS 

12 

13 

10 

15 

Major in English 

6 

6 

39 

45 

Major in reading language arts 

5 

6 

10 

20* 

Years taught 

0-4 years 

37* 

16 

43* 

18 

5-9 years 

33 

33 

36 

27 

10-19 years 

16 

30* 

13 

35* 

20 - 1 - years 

3 

14* 

9 

20* 


*The difference between charter schools and regular public schools is statistically significant at 


the .05 level. 
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IV. Strengthening the NAEP Data and the Data Explorer 

One of the purposes of this exploratory study was to identify ways that the NAEP background 
data could be made more useful. So in this final section, we offer some suggestions based on our 
experience using the NAEP data, not only on this project, but in our past research. 

The main way that researchers access the NAEP data are through the online NAEP Data 
Explorer. It is a powerful tool and, with practice, researchers can conduct a wide variety of 
analyses. However, the interface is not as user-friendly as it could be. Many of the actions 
necessary to conduct certain analyses are not intuitive, and it is only after trial-and-error (or with 
the help of a more-experienced colleague) that one discovers how to retrieve the data or carry- 
out the desired analyses. It would be worthwhile to take the steps necessary to refine the Data 
Explorer, so that research projects could be conducted as intuitively and efficiently as possible. 
Perhaps this could be done by gathering some experienced users of the Data Explorer together 
with programmers who develop the website, to have a brainstorming session on ways to improve 
the interface. We are starting out with a good tool, so there is no need to reinvent it — simply to 
refine it and work through some of the kinks that exist. 

One major difficulty for the user is sorting through the hundreds of background variables listed 
in the Data Explorer to find the few of interest. The majority of background questions listed in 
the Data Explorer were asked in earlier years but are no longer asked. So the researcher must 
scroll through hundreds of variables which are no longer relevant or for which data are no longer 
available. It would be helpful if, once the researcher checks the years of interest, only those 
variables available for that year (or years) show up on the screen. 

The background variables are also not organized as well as they could be. The responses from 
the school, teacher, and student-level questionnaires are often mixed together. If one is interested 
in time spent on instruction in a certain subject, it is necessary to scroll through dozens of 
variables to find that question, unless one happens to know that it is under the heading of 
“Instructional Content and Practice” and then under the subheading “Classroom Management” 
(not at all intuitive). One option might be to add a search bar to the program, so that the user 
could type in a key word and find relevant variables that way. 

We realize that NCES regularly reviews the background variables, but it still seems that there is 
some redundancy and areas where useful information is missing. Some questions may be out of 
date, such as whether a student has a set of encyclopedias at home. Other background variables 
seem overly detailed, such as some of those in the area of teacher professional development. 
There are also old questions that could be reinstated, such as those regarding school climate — 
school safety, parent involvement, gang activity, discipline — that are still clearly relevant and of 
interest to education stakeholders. 

Einally, we ran into a specific problem when trying to research charter schools, related to the 
TUDA data. Due to a policy change in 2009, charter schools within the geographic boundaries of 
an urban district, but operated independently of the district and not included in the district’s AYP 
report under NCEB, are no longer included in the NAEP results for TUDA districts. School 
districts vary in terms of whether the charter schools within their boundaries are included in the 
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district’s AYP results. While we understand why districts would not want charter schools 
included in their results, it makes it impossible to compare charter and non-charter performance 
within most major cities where large numbers of charter schools are located. In this study we 
found a clear pattern of charter schools outperforming regular schools in the four cities that had 
sufficient data. It would be desirable to find a way to include an option for including non-district- 
govemed charter schools in an urban district’s results, solely for research purposes (not for 
general TUDA reporting). After all, charter schools are likely to continue to grow in importance 
and be the focus of research. These schools are serving ever-larger numbers and percentages of 
students — particularly minority and low income students — and the large urban districts are 
where they should be studied. 
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